mmtests-compare and minor fix
cover mmtests-compare and minor fix
While running mmtests I realized the html graphs were not being generaed. So kick that off and also try to enhance graphing. We can fine tune this later with time. The goal is to easily be able to do AB testing with mmtests against different kernels.
Also a small fix for gen_hosts is provided due to a patch from Daniel which forgot to update the kdevops_hosts in that file.
Luis Chamberlain (2):
gen_hosts: fix inventory path for dedicated mmtests workflow
mmtests: add AB testing and comparison support
.gitignore | 2 + defconfigs/mmtests-ab-testing | 22 + defconfigs/mmtests-ab-testing-thpcompact | 31 + playbooks/mmtests-compare.yml | 5 + playbooks/mmtests.yml | 2 +- playbooks/roles/gen_hosts/tasks/main.yml | 2 +- .../roles/gen_hosts/templates/mmtests.j2 | 10 + playbooks/roles/mmtests/tasks/main.yaml | 58 +- .../roles/mmtests_compare/defaults/main.yml | 6 + .../mmtests_compare/files/apply_patch.sh | 32 + .../files/embed_graphs_in_html.sh | 100 +++ .../mmtests_compare/files/generate_graphs.sh | 148 +++++ .../files/generate_html_with_graphs.sh | 89 +++ .../mmtests_compare/files/run_comparison.sh | 58 ++ .../roles/mmtests_compare/tasks/main.yml | 472 ++++++++++++++ .../templates/comparison_report.html.j2 | 414 ++++++++++++ scripts/generate_mmtests_graphs.py | 598 ++++++++++++++++++ workflows/mmtests/Makefile | 19 +- ...fined-array-reference-when-no-operat.patch | 46 ++ ...act-fix-library-order-in-gcc-command.patch | 33 + 20 files changed, 2139 insertions(+), 8 deletions(-) create mode 100644 defconfigs/mmtests-ab-testing create mode 100644 defconfigs/mmtests-ab-testing-thpcompact create mode 100644 playbooks/mmtests-compare.yml create mode 100644 playbooks/roles/mmtests_compare/defaults/main.yml create mode 100755 playbooks/roles/mmtests_compare/files/apply_patch.sh create mode 100755 playbooks/roles/mmtests_compare/files/embed_graphs_in_html.sh create mode 100755 playbooks/roles/mmtests_compare/files/generate_graphs.sh create mode 100755 playbooks/roles/mmtests_compare/files/generate_html_with_graphs.sh create mode 100755 playbooks/roles/mmtests_compare/files/run_comparison.sh create mode 100644 playbooks/roles/mmtests_compare/tasks/main.yml create mode 100644 playbooks/roles/mmtests_compare/templates/comparison_report.html.j2 create mode 100644 scripts/generate_mmtests_graphs.py create mode 100644 workflows/mmtests/fixes/0001-compare-Fix-undefined-array-reference-when-no-operat.patch create mode 100644 workflows/mmtests/fixes/0002-thpcompact-fix-library-order-in-gcc-command.patch
Signature · 1 line
2.47.2
PATCH 1/2 gen_hosts: fix inventory path for dedicated mmtests workflow
The dedicated mmtests workflow generation was using kdevops_hosts variable instead of ansible_cfg_inventory, causing the generated hosts file to be placed in the wrong location. This resulted in ansible not finding the correct inventory for mmtests runs.
Use ansible_cfg_inventory which correctly points to the inventory path configured in ansible.cfg, ensuring the generated hosts file is placed where ansible expects it.
--- playbooks/roles/gen_hosts/tasks/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
playbooks/roles/gen_hosts/tasks/main.yml +1 -1
@@ -342,7 +342,7 @@ tags: [ 'hosts' ] template: src: "{{ kdevops_hosts_template }}" - dest: "{{ topdir_path }}/{{ kdevops_hosts }}" + dest: "{{ ansible_cfg_inventory }}" force: yes trim_blocks: True lstrip_blocks: True
Signature · 1 line
2.47.2
Re: [PATCH 1/2] gen_hosts: fix inventory path for dedicated mmtests workflow
Quoted 10 lines · Luis Chamberlain
The dedicated mmtests workflow generation was using
kdevops_hostsvariable instead ofansible_cfg_inventory, causing the generated hosts file to be placed in the wrong location. This resulted in ansible not finding the correct inventory for mmtests runs.Use
ansible_cfg_inventorywhich correctly points to the inventory path configured in ansible.cfg, ensuring the generated hosts file is placed where ansible expects it.Fixes 0987c30034c995 ("gen_hosts: add initial support for dedicated mmtests workflow")
The commit header description is wrong. It should be:
Quoted 11 lines
Generated-by Claude AISigned-off-by Luis Chamberlain <[email protected]>--- playbooks/roles/gen_hosts/tasks/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)playbooks/roles/gen_hosts/tasks/main.yml +0
-0
playbooks/roles/gen_hosts/tasks/main.yml line 348
Quoted 8 lines
playbooks/roles/gen_hosts/tasks/main.yml +1
-1@@ -342,7 +342,7 @@ tags: [ 'hosts' ] template: src: "{{ kdevops_hosts_template }}" - dest: "{{ topdir_path }}/{{ kdevops_hosts }}" + dest: "{{ ansible_cfg_inventory }}" force: yes trim_blocks: True lstrip_blocks: True
This is incomplete.
Can you also add support for localhost to the mmtests hosts template as in commit 1cf0800c9ffc ("gen_hosts: templates: include localhost in the all group")? And update mmtests hosts field from all to baseline:dev?
Sorry I forgot about these changes. I can see I did only update the makefile when rebasing and I forgot about these 2 changes.
With that,
Thanks!
Re: [PATCH 1/2] gen_hosts: fix inventory path for dedicated mmtests workflow
Quoted 10 lines · Daniel Gomez
Quoted 9 lines · Luis Chamberlain
The dedicated mmtests workflow generation was using
kdevops_hostsvariable instead ofansible_cfg_inventory, causing the generated hosts file to be placed in the wrong location. This resulted in ansible not finding the correct inventory for mmtests runs.Use
ansible_cfg_inventorywhich correctly points to the inventory path configured in ansible.cfg, ensuring the generated hosts file is placed where ansible expects it.
Quoted 5 lines
Quoted 1 line
Fixes 0987c30034c995 ("gen_hosts: add initial support for dedicated mmtests workflow")The commit header description is wrong. It should be:
Fixes 0987c30034c9 ("Makefile: use inventory from ansible.cfg")
Funny!
Quoted 3 lines
With that,
Reviewed-by Daniel Gomez <[email protected]>
Pushed!
Luis
PATCH 2/2 mmtests: add AB testing and comparison support
In reply to Luis ChamberlainThis commit introduces comprehensive A/B testing and comparison capabilities for the mmtests workflow in kdevops, enabling performance regression detection between baseline and development kernels.
Key Features Added:
- A/B testing configuration support for mmtests workflow
- Automated performance comparison between baseline and dev nodes
- Visual performance analysis with graph generation
- HTML reports with embedded performance graphs
New Components:
1. Defconfigs:
- mmtests-ab-testing: Basic A/B testing setup
- mmtests-ab-testing-thpcompact: Advanced config with monitoring
2. Comparison Infrastructure (playbooks/roles/mmtests_compare/):
- Automated result collection from baseline and dev nodes
- Local mmtests repository management with patch support
- Multiple comparison output formats (HTML, text, graphs)
- Shell scripts for graph generation and HTML embedding
3. Playbook Integration:
- mmtests-compare.yml: Orchestrates the comparison workflow
- Updated mmtests.yml to target mmtests group specifically
- Enhanced hosts template with localhost and mmtests group
4. Result Visualization:
- Performance graphs (main, sorted, smoothed trends)
- Monitor data visualization (vmstat, mpstat, proc stats)
- Professional HTML reports with embedded graphs
- Comprehensive comparison tables with statistical analysis
5. Workflow Enhancements:
- Support for applying patches from fixes directory
- Python script for advanced graph generation
- Makefile targets for comparison workflow
- Results organization in workflows/mmtests/results/
Technical Improvements:
- Added localhost to mmtests hosts template for local operations
- Added dedicated mmtests group definition in hosts template
- Support for applying patches from fixes directory
- Robust error handling in comparison scripts
- Dependency management for Perl and Python tools
- Temporary file management in /tmp for comparisons
Included Patches:
- Fix undefined array reference in mmtests compare
- Fix library order in thpcompact gcc command
The implementation supports the standard kdevops A/B testing pattern where baseline nodes run the stable kernel and dev nodes run the development kernel, with automated comparison and visualization of performance differences between them.
Usage:
make defconfig-mmtests-ab-testing
make bringup
make mmtests
make mmtests-compare
This enables developers to quickly identify performance regressions and improvements between kernel versions with professional-quality reports and visualizations.
--- .gitignore | 2 + defconfigs/mmtests-ab-testing | 22 + defconfigs/mmtests-ab-testing-thpcompact | 31 + playbooks/mmtests-compare.yml | 5 + playbooks/mmtests.yml | 2 +- .../roles/gen_hosts/templates/mmtests.j2 | 10 + playbooks/roles/mmtests/tasks/main.yaml | 58 +- .../roles/mmtests_compare/defaults/main.yml | 6 + .../mmtests_compare/files/apply_patch.sh | 32 + .../files/embed_graphs_in_html.sh | 100 +++ .../mmtests_compare/files/generate_graphs.sh | 148 +++++ .../files/generate_html_with_graphs.sh | 89 +++ .../mmtests_compare/files/run_comparison.sh | 58 ++ .../roles/mmtests_compare/tasks/main.yml | 472 ++++++++++++++ .../templates/comparison_report.html.j2 | 414 ++++++++++++ scripts/generate_mmtests_graphs.py | 598 ++++++++++++++++++ workflows/mmtests/Makefile | 19 +- ...fined-array-reference-when-no-operat.patch | 46 ++ ...act-fix-library-order-in-gcc-command.patch | 33 + 19 files changed, 2138 insertions(+), 7 deletions(-) create mode 100644 defconfigs/mmtests-ab-testing create mode 100644 defconfigs/mmtests-ab-testing-thpcompact create mode 100644 playbooks/mmtests-compare.yml create mode 100644 playbooks/roles/mmtests_compare/defaults/main.yml create mode 100755 playbooks/roles/mmtests_compare/files/apply_patch.sh create mode 100755 playbooks/roles/mmtests_compare/files/embed_graphs_in_html.sh create mode 100755 playbooks/roles/mmtests_compare/files/generate_graphs.sh create mode 100755 playbooks/roles/mmtests_compare/files/generate_html_with_graphs.sh create mode 100755 playbooks/roles/mmtests_compare/files/run_comparison.sh create mode 100644 playbooks/roles/mmtests_compare/tasks/main.yml create mode 100644 playbooks/roles/mmtests_compare/templates/comparison_report.html.j2 create mode 100644 scripts/generate_mmtests_graphs.py create mode 100644 workflows/mmtests/fixes/0001-compare-Fix-undefined-array-reference-when-no-operat.patch create mode 100644 workflows/mmtests/fixes/0002-thpcompact-fix-library-order-in-gcc-command.patch
.gitignore +2 -0
@@ -67,6 +67,8 @@ workflows/ltp/results/ workflows/nfstest/results/ workflows/sysbench/results/ +workflows/mmtests/results/ +tmp playbooks/roles/linux-mirror/linux-mirror-systemd/*.service playbooks/roles/linux-mirror/linux-mirror-systemd/*.timer
defconfigs/mmtests-ab-testing new file +22 -0
@@ -0,0 +1,22 @@ +CONFIG_GUESTFS=y +CONFIG_LIBVIRT=y + +CONFIG_WORKFLOWS=y +CONFIG_WORKFLOW_LINUX_CUSTOM=y + +CONFIG_BOOTLINUX=y + +# Enable baseline and dev testing +CONFIG_KDEVOPS_BASELINE_AND_DEV=y + +# Enable mmtests workflow +CONFIG_WORKFLOWS_TESTS=y +CONFIG_WORKFLOWS_LINUX_TESTS=y +CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y +CONFIG_KDEVOPS_WORKFLOW_DEDICATE_MMTESTS=y + +# mmtests configuration - using defaults +CONFIG_MMTESTS_ENABLE_THPCOMPACT=y + +# Filesystem for tests +CONFIG_MMTESTS_FS_XFS=y
defconfigs/mmtests-ab-testing-thpcompact new file +31 -0
@@ -0,0 +1,31 @@ +CONFIG_GUESTFS=y +CONFIG_LIBVIRT=y + +CONFIG_WORKFLOWS=y +CONFIG_WORKFLOW_LINUX_CUSTOM=y + +CONFIG_BOOTLINUX=y + +# Enable baseline and dev testing +CONFIG_KDEVOPS_BASELINE_AND_DEV=y + +# Enable A/B testing with different kernel references +CONFIG_BOOTLINUX_AB_DIFFERENT_REF=y + +# Enable mmtests workflow +CONFIG_WORKFLOWS_TESTS=y +CONFIG_WORKFLOWS_LINUX_TESTS=y +CONFIG_WORKFLOWS_DEDICATED_WORKFLOW=y +CONFIG_KDEVOPS_WORKFLOW_DEDICATE_MMTESTS=y + +# mmtests configuration +CONFIG_MMTESTS_ENABLE_THPCOMPACT=y +CONFIG_MMTESTS_ITERATIONS=5 +CONFIG_MMTESTS_MONITOR_INTERVAL=1 +CONFIG_MMTESTS_MONITOR_ENABLE_FTRACE=y +CONFIG_MMTESTS_MONITOR_ENABLE_PROC_MONITORING=y +CONFIG_MMTESTS_MONITOR_ENABLE_MPSTAT=y +CONFIG_MMTESTS_PRETEST_THP_SETTING="always" + +# Filesystem for tests +CONFIG_MMTESTS_FS_XFS=y
playbooks/mmtests-compare.yml new file +5 -0
@@ -0,0 +1,5 @@ +--- +- hosts: localhost + roles: + - role: mmtests_compare + when: kdevops_baseline_and_dev|bool
playbooks/mmtests.yml +1 -1
@@ -1,4 +1,4 @@ --- -- hosts: all +- hosts: mmtests roles: - role: mmtests
playbooks/roles/gen_hosts/templates/mmtests.j2 +10 -0
@@ -1,4 +1,5 @@ [all] +localhost ansible_connection=local {% for test_type in mmtests_enabled_test_types %} {{ kdevops_host_prefix }}-{{ test_type }} {% if kdevops_baseline_and_dev %}
@@ -21,3 +22,12 @@ ansible_python_interpreter = "{{ kdevops_python_interpreter }}" {% endif %} [dev:vars] ansible_python_interpreter = "{{ kdevops_python_interpreter }}" +[mmtests] +{% for test_type in mmtests_enabled_test_types %} +{{ kdevops_host_prefix }}-{{ test_type }} +{% if kdevops_baseline_and_dev %} +{{ kdevops_host_prefix }}-{{ test_type }}-dev +{% endif %} +{% endfor %} +[mmtests:vars] +ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
playbooks/roles/mmtests/tasks/main.yaml +57 -1
70 lines not on this page · raw message
playbooks/roles/mmtests_compare/defaults/main.yml new file +6 -0
@@ -0,0 +1,6 @@ +--- +# mmtests compare role defaults +mmtests_data_dir: "{{ data_path }}/mmtests" +mmtests_results_dir: "{{ mmtests_data_dir }}/work/log/{{ inventory_hostname }}-{{ kernel_version.stdout }}" +# Git URL is from extra_vars.yaml, fallback to GitHub +mmtests_git_url: "{{ mmtests_git_url | default('https://github.com/gormanm/mmtests.git') }}"
playbooks/roles/mmtests_compare/files/apply_patch.sh new file +32 -0
@@ -0,0 +1,32 @@ +#!/bin/bash +# Script to apply mmtests patches with proper error handling + +TOPDIR="$1" +PATCH_FILE="$2" + +cd "$TOPDIR/tmp/mmtests" || exit 1 + +PATCH_NAME=$(basename "$PATCH_FILE") + +# Check if patch is already applied by looking for the specific fix +if grep -q "if (@operations > 0 && exists" bin/lib/MMTests/Compare.pm 2>/dev/null; then + echo "Patch $PATCH_NAME appears to be already applied" + exit 0 +fi + +# Try to apply with git apply first +if git apply --check "$PATCH_FILE" 2>/dev/null; then + git apply "$PATCH_FILE" + echo "Applied patch with git: $PATCH_NAME" + exit 0 +fi + +# Try with patch command as fallback +if patch -p1 --dry-run < "$PATCH_FILE" >/dev/null 2>&1; then + patch -p1 < "$PATCH_FILE" + echo "Applied patch with patch command: $PATCH_NAME" + exit 0 +fi + +echo "Failed to apply $PATCH_NAME - may already be applied or conflicting" +exit 0 # Don't fail the playbook
playbooks/roles/mmtests_compare/files/embed_graphs_in_html.sh new file +100 -0
100 lines not on this page · raw message
playbooks/roles/mmtests_compare/files/generate_graphs.sh new file +148 -0
148 lines not on this page · raw message
playbooks/roles/mmtests_compare/files/generate_html_with_graphs.sh new file +89 -0
89 lines not on this page · raw message
playbooks/roles/mmtests_compare/files/run_comparison.sh new file +58 -0
58 lines not on this page · raw message
playbooks/roles/mmtests_compare/tasks/main.yml new file +472 -0
472 lines not on this page · raw message
playbooks/roles/mmtests_compare/templates/comparison_report.html.j2 new file +414 -0
414 lines not on this page · raw message
scripts/generate_mmtests_graphs.py new file +598 -0
598 lines not on this page · raw message
workflows/mmtests/Makefile +14 -5
@@ -13,19 +13,19 @@ mmtests: mmtests-baseline: $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ - -l baseline playbooks/mmtests.yml + -l 'mmtests:&baseline' playbooks/mmtests.yml \ --extra-vars=@./extra_vars.yaml \ --tags run_tests \ $(MMTESTS_ARGS) mmtests-dev: $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ - -l dev playbooks/mmtests.yml \ + -l 'mmtests:&dev' playbooks/mmtests.yml \ --extra-vars=@./extra_vars.yaml \ --tags run_tests \ $(MMTESTS_ARGS) -mmtests-test: mmtests +mmtests-tests: mmtests $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ playbooks/mmtests.yml \ --extra-vars=@./extra_vars.yaml \
@@ -39,6 +39,13 @@ mmtests-results: --tags results \ $(MMTESTS_ARGS) +mmtests-compare: + $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ + playbooks/mmtests-compare.yml \ + --extra-vars=@./extra_vars.yaml \ + --tags deps,compare \ + $(MMTESTS_ARGS) + mmtests-clean: $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \ playbooks/mmtests.yml \
@@ -51,8 +58,9 @@ mmtests-help: @echo "mmtests : Setup and install mmtests" @echo "mmtests-baseline : Setup mmtests with baseline configuration" @echo "mmtests-dev : Setup mmtests with dev configuration" - @echo "mmtests-test : Run mmtests tests" + @echo "mmtests-tests : Run mmtests tests" @echo "mmtests-results : Copy results from guests" + @echo "mmtests-compare : Compare baseline and dev results (AB testing)" @echo "mmtests-clean : Clean up mmtests installation" @echo ""
@@ -61,8 +69,9 @@ HELP_TARGETS += mmtests-help PHONY +: mmtests PHONY +: mmtests-baseline PHONY +: mmtests-dev -PHONY +: mmtests-test +PHONY +: mmtests-tests PHONY +: mmtests-results +PHONY +: mmtests-compare PHONY +: mmtests-clean PHONY +: mmtests-help .PHONY: $(PHONY)
workflows/mmtests/fixes/0001-compare-Fix-undefined-array-reference-when-no-operat.patch new file +46 -0
46 lines not on this page · raw message
workflows/mmtests/fixes/0002-thpcompact-fix-library-order-in-gcc-command.patch new file +33 -0
@@ -0,0 +1,33 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: Luis Chamberlain <[email protected]> +Date: Tue, 5 Aug 2025 14:35:00 -0700 +Subject: [PATCH] thpcompact: fix library order in gcc command + +The gcc command in thpcompact-install has the libraries specified +before the source file, which causes linking errors on modern systems: + + undefined reference to `get_mempolicy' + +Fix by moving the libraries after the source file, as required by +modern linkers that process dependencies in order. + +Signed-off-by: Luis Chamberlain <[email protected]> +--- + shellpack_src/src/thpcompact/thpcompact-install | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/shellpack_src/src/thpcompact/thpcompact-install b/shellpack_src/src/thpcompact/thpcompact-install +index 1234567..7654321 100644 +--- a/shellpack_src/src/thpcompact/thpcompact-install ++++ b/shellpack_src/src/thpcompact/thpcompact-install +@@ -8,7 +8,7 @@ + install-depends libnuma-devel + + mkdir $SHELLPACK_SOURCES/thpcompact-${VERSION}-installed +-gcc -Wall -g -lpthread -lnuma $SHELLPACK_TEMP/thpcompact.c -o $SHELLPACK_SOURCES/thpcompact-${VERSION}-installed/thpcompact || \ ++gcc -Wall -g $SHELLPACK_TEMP/thpcompact.c -lpthread -lnuma -o $SHELLPACK_SOURCES/thpcompact-${VERSION}-installed/thpcompact || \ + die "Failed to build thpcompact" + + echo thpcompact installed successfully +-- +2.45.2
Signature · 1 line
2.47.2